Ac 2007-2366: Capstone Design Projects with Industry: Using Rubrics to Assess Student Design Reports

The benefits of company sponsored capstone design projects, both to academia and to industry, have been well established. Specific benefits to students include the broadening of their engineering skills, the required interaction with practicing engineers, the strengthening of teaming skills by working in design groups, the development of communication skills with required oral and written reports, and the experiences of project management. At the authors’ institution these projects are “owned and managed” by the student teams with company contacts providing appropriate data and information and with faculty serving as advisors only. The authors have developed and improved these student/industry interactions over the last few years with over 130 students working with about 30 different companies each year. ABET 2000 requires that graduates demonstrate the ability to design a system, component or process to meet a given need and this capstone design course is a natural place to assess whether or not these outcomes are met. Outcomes may be assessed by direct or indirect methods. Direct measures of student outcomes are based on student work, and for capstone design courses a natural work product to examine is the design written report. Typically, performance criteria are established and then rubrics are written to insure the consistency of the assessment. The purpose of this paper is to show how such rubrics were developed for senior mechanical engineering design reports and then how they were used by three different groups: the course instructors, other departmental faculty, and outside engineering practitioners. Each of these three groups was given the same set of design reports and then was asked to evaluate the reports by using specifically these scoring rubrics. This paper details the performance criteria, shows the rubrics used, and then reports on the consistency in scoring between these groups. Introduction Capstone Design is defined as those senior projects which attempt to provide a culminating experience to students’ undergraduate engineering education. Traditionally these projects provide team experiences in utilizing engineering concepts learned in previous courses to provide solutions to real world design problems. This general philosophy is followed in the authors’ department but with the following requirements: 1. all project problems are provided by external industrial clients, 2. formal design methodologies must be followed, 3. economics must be considered in design decisions, 4. the improvement of teaming skills must be emphasized, and 5. all design solutions must be communicated through both oral and written reports. The provision of project problems by external industrial clients is not a simple undertaking but is possible with industrial/academic relationships and partnerships that are fostered over time 1 . These partnerships are essential for technological development, regardless of the discipline. At the authors’ institution student design projects with industry are performed as part of course requirements for Engineering Design and for Machine Design, both 4-credit-hour courses. In each course the design project represents 50% of the course grade and the enrollment is approximately 130. The students work in groups of 3 to 5 and are normally expected to spend approximately 4-6 hours per person per week on their projects. Because these courses are taught in two consecutive quarters, the projects last approximately 20 weeks. The teams are required to meet with their instructors weekly and submit written status reports. Formalized design methodologies are emphasized and required. (See for example Pahl and Beitz 2 or Dekker and Gibson 3 .) At the end of the 10 week Winter Quarter the student teams give interim oral presentations and submit interim written reports to their instructors and to their client companies. Final presentations and written reports are due again at the end of the 10 week Spring Quarter. The presentations and written reports are the only deliverables that participating companies are guaranteed. Because every company receives a final design report, these reports have become logical work products to use to evaluate the students’ performances. Rationale As with all capstone design courses the authors are striving to improve the quality of their design reports and of their courses as a whole. Current methods used in their capstone courses to gather feedback include student evaluations, external client surveys, and the completion of a department mandated feedback sheet used to satisfy accreditation requirements. Although student evaluations and external client surveys contain useful information, it is anecdotal and difficult to use in a systematic way to improve course quality. Therefore the authors have begun to use ABET developed performance criteria, along with appropriate rubrics to gain feedback from both external practitioners and other departmental faculty. The ABET developed performance criteria were selected because they are patterned after the desired design methodology. For example, one of the first actions that students must perform is to understand their client’s problem. The entire team is required to visit the client company without any faculty present. This requires that the team generate questions prior to the visit, that they actively listen to their client’s concerns, and that they restate the problem in its most general form. This strategy for design improvement has the following three major objectives: 1. Develop rubrics that clearly describe our course outcomes so that these rubrics can be used by parties external to the course to provide us with meaningful assessments. 2. Use the rubrics to implement peer assessment of the design reports, which has proved helpful in other areas. 3. Use the rubrics to give to students, as a means of communicating to them more clearly how design reports can be assessed, and therefore what they should emphasize. ABET Criteria ABET Criterion 3, Program Outcomes and Assessment requires that all engineering programs demonstrate that their students possess “(c) an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability.” Consistent with this requirement, the authors’ ME Department developed Performance Criteria which are measurable attributes used to define these outcomes. These are stated as: When given the opportunity, students will: 1. Understand the problem. 2. Develop a product or process design specification that addresses customer/client needs and constraints. 3. Carry out a conceptual design by generating multiple solutions that address the issues above, 4. Evaluate the feasibility of the solutions, and 5. Choose the appropriate solution. 6. Carry out a detail-level design using appropriate design tools and methodologies. 7. Test and refine the implementation until the product or process design specifications are met or exceeded. 8. Document the finished product or process as appropriate for the discipline according to standard practice. The ME Department desired a method for obtaining external feedback to validate that the quality of the program is being maintained. The intent was not to obtain additional opinions on team grades, but rather to obtain indications of whether or not the performance criteria had been demonstrated. Therefore, the authors decided to use rubrics to determine if desired outcomes are being met. These rubrics were developed to distinguish three levels of performance: “Excellent”, “Satisfactory”, and “Needs Improvement”. In addition, Performance Criterion 8 (above) was modified since outside practitioners may not be aware of the departmental standards for documentation. The rubric therefore was changed to emphasize “conclusions and recommendations”. The rubrics that finally were developed to demonstrate these Performance Criteria are listed below and are italicized for emphasis. 1. Understand the problem Excellent: The students presented the problem in its most general form and the context for the problem was well defined. Satisfactory: The students presented a problem and gave a general context. Needs Improvement: The students presented a solution rather than a problem and/or the context was not clearly defined. 2. Develop a Product Design Specification (PDS) that addresses client needs and constraints Excellent: The PDS was specific and included both goals and quantifiable, measurable constraints. Applicable codes and standards were appropriately referenced. Satisfactory: The PDS was specific and measurable but a clear distinction was not made between goals and constraints. Applicable codes and standards were mentioned but were not fully referenced. Needs Improvement: The PDS was not specific and measurable. Applicable codes and standards were not referenced. 3. Generate multiple solutions Excellent: The students presented at least three feasible solutions. Satisfactory: The students presented multiple solutions but not all solutions were feasible. Needs Improvement: The students presented only one feasible solution. 4. Evaluate feasibility of solutions Excellent: The students used acceptable techniques such as Pugh Concept Selection or decision matrices to evaluate their design concepts. A rationale was given for every rating. Satisfactory: The students used acceptable techniques to evaluate competing solutions, but no rationale was given for ratings. Needs Improvement: The students do not have a systematic method for evaluating design decisions. 5. Choose an appropriate solution Excellent: The solution was clearly documented based on the authors’ feasibility and economic evaluation. Satisfactory: The solution was not fully documented. Needs Improvement: The students had an insufficient rational basis for their